Combining vision and functional neuroimaging data to detect the early stages of Cognitive Fatigue

Analyzing human activity is a basic component of any system, be it biological or artificial, that aims to predict future behavior. Tracking and recognizing voluntary and involuntary action traits are basic endeavors for artificial vision systems that aim to predict cognitive fatigue, a major cause for road and workplace accidents. In this work, we developed a vision system to detect the early onset of fatigue. In collaboration with the Magnetoencephalography Lab at MIT, we collected synchronous brain (MEG) and behavioral (high-speed camera) data from 14 subjects in a 3-hour task that was designed to induce cognitive exhaustion. We derived a set of 8 eye-movement and 6 head-movement features and trained classifiers for two classes (fatigue, non-fatigue) and three classes (fatigue, transition stage, non-fatigue). We trained Random Forest, KNearest Neighbor, and Support Vector Machines classifiers, first and achieved average test accuracies of 98%, 97%, 92% (two classes) and 92%, 90%, 87% (three classes) respectively. To further validate our models, we used the alpha band power in the MEG data as the neural indicator of fatigue. A regression analysis between the camera-based features and the alpha band power revealed an average ˆ2 = 0.59. Here, we also propose a new method to detect the early stages of fatigue by using the classification error as our behavioral marker. Specifically, we found that the accuracy of the classifiers was higher when the distance between the time intervals of labels for non-fatigue and fatigue was larger; We estimated the total number of the mis-classified fatigue and non-fatigue data points in a sliding window: the fatigue (non-fatigue) number was high (low) in the beginning -non-fatigue stageand became lower (higher) with time, signifying clear periods of fatigue and non-fatigue. We also observed a sharp change in the labels from non-fatigue to fatigue after 40-50 minutes, which can be attributable to detecting early stages of fatigue. Our results are promising in terms of designing a fully automated system that can predict ones effective operation range, based on behavioral and neurophysiological cues. Defense Committee: K. Michmizos (chair), D. Metaxas, A. Elgammal